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Abstract 


Within the overall framework of design-based research, this paper reports on a study 
that focused on evaluating an online training course for online instructors. This 
intervention was designed as a possible solution to the problem facing many higher 
education institutions of how to provide quality, accessible training for mostly part-time 
instructors who are making the transition to teaching online. The research project 
explored whether the training course had any impact on the participants’ later teaching 
practice. The major outcome of this research study is the identification of design 
principles that can be used by other researchers and practitioners designing online 
instructor training. 

Keywords: Online teaching; adjunct faculty; design-based research; faculty training; 
online teaching professional development 
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Introduction 


Within the overall framework of design-based research (Barab & Squire, 2004; Design- 
Based Research Collective, 2003; Sandoval & Bell, 2004), this paper reports on a 
research project that focused on evaluating a training course for online instructors: 
MarylandOnline’s Certificate for Online Adjunct Teaching (COAT) course. COAT was 
designed as a possible solution to the problem facing some higher education institutions 
of how to provide quality, accessible training for part-time instructors who are making 
the transition to teaching online. COAT is a nine-week, online, instructor-led course that 
was designed to prepare adjunct faculty to teach their first online course. Although 
designed specifically for adjunct faculty with no online teaching experience, COAT 
participants were found to be more diverse than originally planned for with full-time 
faculty, administrators, and instructors with extensive prior online teaching experience 
enrolling in the course (Shattuck, 2013). This article reports the findings from a study 
that explored the impact of the COAT course on the participants’ later teaching practice. 

Design-Based Research 

Design-based research (DBR), also called design experiments (Brown, 1992), design 
research (Collins, Joseph, & Bielaczyc, 2004), and educational design research 
(McKenney & Reeves, 2012), has generated increasing interest among educational 
researchers in the last decade (Anderson & Shattuck, 2012). Wang and Hannafin (2005) 
defined DBR as “A systematic but flexible methodology aimed to improve educational 
practices through iterative analysis, design, development, and implementation, based 
on collaboration among researchers and practitioners in real-world settings and leading 
to contextually-sensitive design principles and theories” (pp. 6-7). DBR positions 
researchers with practitioners as part of a team that works together, usually over an 
extended period of time, to provide a solution(s) to a practical problem that faces a 
specific educational context. DBR studies use the term intervention to denote the object, 
activity, or process that is designed as a possible solution to address the identified 
problem. McKenney and Reeves (2012) identified intervention as a broad term used “to 
encompass the different kinds of solutions that are designed” (p. 14); these solutions 
include educational products, processes, programs, and policies. This study identified 
the COAT course as the intervention that was developed as a potential solution to the 
perceived need for better high quality training for online adjunct faculty. 

DBR projects can span many years with multiple research cycles that focus on the 
iterative stages of the project analysis, design, development, implementation, and 
evaluation phases. In order to clearly explain how the research reported in this paper 
was positioned within a collaborative, ongoing DBR project, it is helpful to use 
McKenney and Reeves’ (2012) generic model for design research (GMDR) to provide an 
outline of the COAT project phases. The GMDR consists of three main phases, analysis 
and exploration, design and construction, and evaluation and reflection, that lead to the 
two eventual outputs of increased theoretical understanding and effective intervention 
maturation. The three phases of analysis/exploration, design/construction, and 
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evaluation/reflection interact with ongoing practice as the intervention is adopted, 
enacted, and sustained (implementation) in a particular educational setting while 
information about the intervention is disseminated and diffused to a wider audience. 


Using the GMDR to frame the COAT project, this study is situated in phase three: 
evaluation/reflection. Table 1 briefly outlines how the COAT project aligned with the 
initial iteration of the three phases of the GMDR with implementation added to the 
second phase in order to include how the COAT project implemented the first iteration 
of the training course after a successful pilot. 

Table 1 


COAT Project and Generic Model for Design Research Phases 


Analysis/Exploration Design/Construction (& 
_ Implementation) _ 

When? Fall 2008-spring 2009 Fall 2009-spring 2012 


What 1. Literature review on 

research/ online teaching roles 

activities? and competencies. 

2. Survey of 37 
Maryland higher 
education institutions. 

3. Interviews with key 
personnel from 17 US 
higher education online 
teaching training 
programs. 


1. Developed, designed, & 
evaluated pilot course. 

2. Ran 11 sections of 
COAT course. 

3. Used results from 
module & end-of-course 
surveys, course 
assignments, & reflection 
journals to make minor 
ongoing changes to 
design. 


Evaluation / Reflection 


Summer 2012-spring 2013 

1. Questionnaire to COAT 
alumni to find out who took 
the COAT course and why. 

2. Focus groups of alumni 
who taught online after 
taking COAT to identify key 
characteristics that 
impacted later practice. 

3. Observations/interviews 
focused on COAT’s impact 
on later practice. 


In DBR, the content, structure, and instructional approaches of an intervention are first 
identified in the analysis and exploration phase of a design project through a literature 
review and the input of experts and practitioners. This information is then used to 
design the first iteration of the intervention. A preliminary literature review is 
conducted with the purpose of identifying draft design principles that have the potential 
to address the problem the intervention is being designed to solve. In the COAT project, 
the draft design guidelines included what content, structure, and instructional 
approaches might best be used to address the training and experiential needs of adjunct 
faculty who are making the transition to teaching online. 

A preliminary literature review was conducted in fall 2008 to spring 2009 and the 
results were disseminated (Dubins & Graham, 2009). Findings from that literature 
review, combined with interviews with key personnel from 17 US training courses for 
online instructors and a survey of distance learning administrators and faculty trainers 
at Maryland higher education institutions that offer online courses, informed the 
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development of the first iteration of the COAT course. This course placed instructors as 
students within a paced, asynchronous online course that was structured to be similar to 
the type of online courses they may teach in higher education institutions. The results 
and discussion of the design/construction phase which included a pilot run of the 
course are published elsewhere (Shattuck, Dubins, & Zilberman, 2011). This paper 
focuses on the initial evaluation/reflection phase of the COAT project. 

DBR has proved to be an effective approach for other research projects focused on the 
design and evaluation processes of instructor training programs and initiatives (Dede, 
Ketelhut, Whitehouse, Breit, & McCloskey, 2009). Ketelhut, McCloskey, Dede, Breit, 
and Whitehouse (2006) highlighted the importance of, and tension between, the dual 
goals of asking program evaluation questions about the effectiveness of online teacher 
professional development (oTPD) programs and asking empirical research questions 
about the impact of oTPD programs, and they identified DBR as a promising approach 
to address both of these goals. A comparison of DBR to other methodological 
approaches reinforces the choice of DBR as the appropriate approach for this study. 

Collins, Joseph, and Bielaczyc (2004) compared DBR to three types of educational 
research approaches: laboratory and training studies; ethnographic research; and large- 
scale studies. They argued that laboratory and training settings do not account for 
multiple variables, multiple participants’ expertise, and “the messy situations that 
characterize real life learning” (Collins et al., 2004, p. 20); ethnographic research 
describes in detail what and why relationships and events occur, but it does not try to 
change practice; and large-scale studies “do not provide the kind of detailed picture 
needed to guide the refinement of a design” (p. 21). 

Action research has similarities with DBR in terms of collaboration, researchers having 
multiple roles, and reflection on practice. Typical action research, as opposed to critical 
action research, positions the teacher-as-researcher conducting “a form of disciplined 
inquiry, in which a personal attempt is made to understand, improve, and reform 
practice” (Cohen, Manion, & Morrison, 2007, p. 297). Action research can be 
collaborative, but the emphasis is on reflective research to inform individual practice at 
the local level. DBR, in contrast, is always collaborative with a focus on the generation of 
design principles in an evaluation/reflection phase where reflection "involves active and 
thoughtful consideration of what has come together in both research and development 
(including theoretical inputs, empirical findings, and subjective reactions) with the aim 
of producing new (theoretical) understanding" (McKenney & Reeves, 2012, p. 151). 
Developing design principles is part of an ongoing DBR process that may eventually 
lead to theoretical understanding: 

The outcomes of design-based research are a set of 
design principles or guidelines derived empirically and 
richly described, which can be implemented by others 
interested in studying similar settings and concerns. 

While the ultimate objective is the development of 
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theory, this might only occur after long-term 
engagement and multiple design investigations. (Amiel 
& Reeves, 2008, p. 35) 

One of the goals of the study reported in this paper was to use the detailed data that 
were collected and analyzed from the evaluation of the first iteration of the COAT course 
to articulate design principles that are relevant to other distance learning professionals 
and that are transferable to similar contexts. 

Purpose and Research Questions 

The purpose of this research study was to evaluate whether the content, structure, and 
instructional approaches of the COAT course effectively helped prepare higher 
education instructors to teach online, and, through reflection, to extract design 
principles that could prove useful for other researchers and practitioners working in the 
field of online instructor training. The research model used to evaluate the COAT course 
was Guskey’s (2000) model of five critical levels of professional development 
evaluation. This study collected and analyzed data focused on evaluating Level 4: 
participants’ use of new knowledge and skills. 

Guskey (2000) argued that most evaluations of professional development occur at Level 
1 (initial participant reactions) as it is the easiest level to assess. However, data gathered 
from the lower levels are not informative for measuring the impact of training on 
subsequent practice. Guskey highlighted that it is challenging to make a connection 
between teaching practice and earlier training experiences: 

Educators work in complex environments where 
multiple factors affect their behaviors. Changes in 
leadership, occurrences in one's personal life, other 
learning opportunities, or changes in professional 
assignment could alter participants’ behaviors and 
activities quite apart from the influence of professional 
development. Isolating the professional development 
experience as the true cause of change in practice is a 
challenging aspect in any evaluation effort. (2000, p. 

187) 

This study aimed to meet this challenge through utilizing mixed research methods that 
included most of the ways Guskey identified for gathering information at Level 4. His 
methods included: questionnaires; interviews with participants and their supervisors; 
participant reflections; participant portfolios; direct observations; and video or audio 
tapes. The research methods used in the doctoral thesis (Shattuck, 2013) in which this 
study was conducted included a questionnaire, participant reflections in focus groups, 
analysis of archived online courses, and interviews. This paper reports on the second 
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phase of the doctoral study: online, asynchronous focus groups in which the following 
research questions were explored: 

1. Can a sample of alumni who taught online after completing the COAT 
course identify any elements (content, structure, instructional approaches, 
etc.) of the COAT course as being notably important in helping them teach 
their subsequent online course(s)? If yes, which elements? 

2. Do they identify any elements as being unimportant or even misleading in 
informing their subsequent online teaching practice? If yes, which 
elements? 

Research Paradigm 

This study was framed within an interpretivist paradigm which considers that “a 
primary aim of social science is to understand what people mean and intend by what 
they say and do and to locate those understandings within the historical, cultural, 
institutional, and immediate situational contexts that shape them” (Moss et al., 2009, p. 
501). Denzin and Lincoln (2011) state that an interpretivist paradigm “assumes a 
relativist ontology (there are multiple realities), a subjectivist epistemology (knower and 
respondent co-create understandings), and a naturalistic (in the natural world) set of 
methodological procedures” (p. 13). Crotty (1998) differentiated between creating 
understandings, a subjectivist epistemology that sees meaning as being created by 
individuals, and constructing understandings, a constructionist epistemology that 
considers that people construct meaning together in relation to their engagement with 
their human world. This DBR study, with its focus on generating knowledge about a 
training course from the subsequent activities and reflections of COAT participants, 
operated within a social constructionist epistemology. According to Koro-Ljungberg, 
Yendol-Hoppey, Smith, and Hayes (2009), educational researchers working within a 
social constructionist perspective have multifaceted, participatory roles; research goals 
to “negotiate and transform the practice” (p. 690); and a view of knowledge as being 
generated from groups of participants. In addition, this study falls within what Bell 
(2004) described as a “folk (emic) research orientation that investigates the manifested 
meaning of an intervention from the point of view of the participants of the research as 
interpreted through their activity and their accounts” (p. 248). This aligns with DBR’s 
characteristics of offering practical solutions to real world problems from both the 
perspectives of the participants and the researchers involved in the design team. 


Research Method 


This study collected data using online, asynchronous, threaded discussion groups as 
focus groups to explore the research questions using online discussion boards within a 
learning management system (LMS). Turney and Pocknee (2005) researched the use of 
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LMS discussion boards for virtual focus groups, and concluded that asynchronous focus 
groups were theoretically sound because they have the potential to meet Krueger’s 
(1988) six criteria for making a group a focus group: involving people, being conducted 
in a series, having relatively homogenous participants who do not know each other, 
being a method of data collection, collecting qualitative data, and constituting a focused 
discussion. 

Nicholas et al. (2010) summarized the advantages to conducting asynchronous focus 
groups: convenient access; no time constraints allows participants to reflect which leads 
to “data depth and richness” (p. 110); participants cannot interrupt each other; 
emotions can be expressed through emoticons and textual clues; no travel or 
transcription costs; and face validity is fostered “due to member checking, as 
participants have continuous access to the data transcript and have ongoing opportunity 
to reflect on their statements to ensure that meaning is sufficiently captured within the 
data" (p. 110). Disadvantages include a lack of visual clues, time commitments required 
of participants, possible technical barriers to participate, and questions about security of 
data. 

An online, asynchronous format was appropriate for the participants of this study for 
the following reasons. All COAT alumni were accustomed to interacting asynchronously 
using a discussion board as this was an integral part of the COAT course, and, as such, 
technical barriers were not expected to be problematic due to the participants’ 
familiarity with these tools. In addition, the lack of visual clues is something that COAT 
alumni are used to dealing with in their roles as online learners and instructors. The 
LMS used to host the focus group discussion boards is a secure site that is password- 
protected, and participants were given pseudonyms to protect their anonymity. The 
convenience of interacting asynchronously meant that geographic and time constraints 
were less likely to impact the feasibility of setting up the groups. Krueger and Casey 
(2009) suggested inviting asynchronous focus group participants “to spend 15-30 
minutes each day for several days as they review the questions and make their 
responses” (p. 178). By considering the time commitment of about 15-30 minutes a day 
over a period of several days, participants were able to gauge whether they had the time 
needed to participate. 

Focus Group Participants and Logistics 

The purposive sample for the focus groups was derived from the respondents to a 
questionnaire sent to all participants who had completed one of 11 COAT course 
sections that ran fall 2010 to spring 2012. All 126 (out of a possible 179) respondents to 
the online questionnaire were sent an invitation to participate in the focus groups if they 
had taught online after completing COAT which led to 24 COAT alumni participating in 
the focus groups. 

Participants were offered a choice of five dates for the focus groups. Two dates were not 
popular which resulted in three separate groups that ran in July, August, and 
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September, 2012: focus group one (FGi), focus group two (FG2), and focus group three 
(FG3). After signing the informed consent agreement, participants were enrolled in the 
LMS focus group site using a numeric identifier to maintain anonymity. The 
researcher’s role in each focus group was to welcome people to the group, facilitate the 
conversations, and provide summaries of the discussions for member-checking. Each 
group was held asynchronously using the discussion board feature of the LMS that had 
hosted all of the COAT courses. Each focus group was held over three days with a 
separate discussion prompt for each day. FGi and FG2 had identical prompts, and 
FG3 ’s Day One prompt was also identical. However, changes were made to the prompts 
for the second and third days of FG3 based on the ongoing data analysis results from 
FG1-2. 


Each focus group was opened a few days early for participants to preview. The focus 
groups were left open for a week after day three finished, so that participants could 
make any changes or additions to their postings before the data collection period closed 
and data analysis began. No changes were made, but three participants (one in each 
focus group) did add a post the day after the third day. The researcher provided 
summaries for each day’s discussion and a final summary of the whole focus group. 
Participants were invited to make changes and corrections to these summaries. Only 
one clarification was suggested, and three participants verified that the summaries had 
captured what was important from their perspectives. 

Focus group participants were not asked to give detailed demographic information 
about themselves, as this may have compromised their anonymity. However, some 
participants did disclose personal details in their introductions which included the 
information that participants held a number of professional roles within education with 
ten people saying they had worked or were currently working as adjunct faculty, five as 
administrators, three as full-time faculty, two as instructional technologists, and seven 
as Kindergarten-i2 th grade (K-12) instructors. These roles were often held 
simultaneously with the K-12 instructors, administrators, and technologists working as 
higher education adjuncts too. Of the nine participants who gave information about how 
many institutions they were currently employed in, seven people worked in only one 
institution with two others identifying that they worked at more than one institution 
simultaneously. The institutions people worked at were varied with ten community 
college, two university, and one K-12 institutional type identified. Six people had not 
taught online prior to taking COAT, and 11 people had prior online teaching experience 
ranging from one course to over ten years. 

Additional information about participants included prior experience as online students 
with 15 (63%) people having taken online courses before participating in COAT and 
three saying they had no online student experience prior to COAT. Participants also 
talked about the subjects they taught online which included accounting, art, astronomy, 
business, child development, communication, computer science, English, health, 
history, medical assisting, research methods, and statistics. 
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Analysis of Research Results 


This study drew on grounded theory techniques to inform data analysis decisions and 
followed Saldana’s (2009) recommendation to approach coding method choices with 
“pragmatic eclecticism” (p. 47) by letting initial data collection and review occur before 
deciding on which coding method(s) to use. Data analysis for the three focus groups was 
ongoing with preliminary analysis beginning after the first focus group ended and 
further analysis continuing through iterative cycles of initial and focused coding which 
informed data collection decisions for subsequent focus groups. Constant comparison 
(Strauss & Corbin, 1998) of new data against previously collected data and existing 
codes against new codes and emerging categories occurred throughout the process. 
Detailed notes were kept in reflection blogs and analytic memos during the focus group 
data collection and analysis period. 

Important Elements 

All 24 focus group participants identified at least one element of COAT that had 
positively influenced their subsequent online teaching practice, and their comments 
were grouped into the category Taking COAT influenced subsequent online teaching 
practice. This category encompassed 138 quotations which were organized into a code 
family (see Figure 1). This code family included one category (placed top center in 
Figure 1); five codes that stated a particular element of taking COAT which influenced 
later practice or a specified impact on later practice that was attributed to taking COAT 
(shown with the relationship is a in Figure 1); one code that was seen as contributing to 
later teaching practice (shown with the relationship contributes to), and 11 subcodes 
(shown with the relationship is part of). Some of the codes are associated with more 
than one code and these relationships are depicted with is associated with relationship 
arrows. The numbers after each code in Figure 1 show the groundedness and density of 
the code. “Groundedness counts the number of links to quotations; density counts the 
links to other codes and memos” (Friese, 2012, p. 140). 


|Vol 14 | N05 


Dec/13 


194 







Using a Design-Based Research Study to Identify Principles for Training Instructors to Teach Online 


Shattuckand Anderson 



A Improving/Informing online 
practice {15-1} 



Figure 1. Concept map of responses to ways in which taking COAT influenced 
subsequent online teaching practice. 


General comments on COAT’s influence on online practice. 

Two codes were populated with general comments that participants made about COAT 
influencing later teaching practice. Eight quotations referred to participants revisiting 
COAT content as reference materials for later teaching practice, and 15 quotations were 
general comments about COAT’s influence on improving or informing later practice. 
These comments ranged from COAT being seen as an essential part of later online 
teaching success; to enhancing existing online teaching practice; to COAT being seen as 
a minor aid to later teaching practice. The majority of quotations (115 out of 138) were 
more specific about what elements of the COAT course influenced later practice and 
these quotations were grouped into four codes which are presented next in conjunction 
with their subcodes. 
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Experiencing being an online student. 

As shown in Figure 1, this code has four subcodes with a combined total of 28 linked 
quotations that were made by 16 (out of 24) focus group participants. To be included in 
this code, quotations had to explicitly highlight that the experience of being positioned 
as a student in the COAT course influenced participants’ subsequent online teaching 
practice. These experiences could be positive or negative. The 11 quotations directly 
linked to the code were all positive comments about experiencing life as an online 
student affecting later online teaching practice. For some participants COAT was their 
first experience as an online student: 

I think participating as a student in the COAT training 
was of utmost importance. As a novice to online 
teaching, I really needed to be a student. Plus 
participating as a student opened my eyes to so many 
things I would have never even considered if I was just 
reading about these topics. 

Others had taken online courses: “I have been an online student before, but this course 
helped me to focus on the student experience a little better because I was taking the 
course as an instructor wanting to provide a better experience for my students.” This 
concept of benefiting from looking at their own teaching practice through the lens of an 
online student was mentioned by multiple participants; for example, 

I could see what instructional techniques worked well, 
and which ones did not. Being in the student role made 
me look at both sides of a situation. Did I as a student 
learn something from the exercise? Would I as an 
instructor get valuable feedback from this exercise. If it 
doesn't meet both criteria, it must be changed. 

This idea of learning from what participants did not like about experiencing life as an 
online student in COAT was detailed more in two of the four subcodes. Three different 
participants described feeling frustrated by parts of their COAT experience. For one, the 
frustration led her to be mindful of students’ possible frustration with the same element 
in her own courses: 

I had not been in the ‘online’ student role for -7 years 
when I took the COAT course. I was not familiar with the 
learning platform used to deliver the COAT course... This 
unfamiliarity gave me an opportunity to become 
frustrated and remember many students will experience 
this when they are in my class. 
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This idea of negative experiences in the COAT course leading to more awareness of their 
own students experiencing the same feeling was also apparent in the quotations linked 
under the subcode Experiencing time commitment; for example, 

I also felt that there was an enormous amount of 
material presented, and was overwhelmed near the end 
of the class with all of the requirements. I actually went 
back to my classes again and rethought some of my 
expectations, juggling assignment positions within the 
time frame. 

Experiencing the COAT course design as a student also led to changes in participants’ 
later practice with eight quotations related to this idea. For example, one participant 
highlighted copying design features from COAT that she liked into her own courses: 

As a result of taking the COAT course I modified my own 
courses to incorporate some of the instructional design 
features from the COAT course... for example, a separate 
button for "Weekly Course Work," and separate folders 
for each week. 

Three participants mentioned that COAT introduced them to features of the LMS they 
had been unaware of which led to them using these features in their own practice. 

COAT was designed to give participants the experience of being an online student in a 
paced, facilitator-led, cohort-based course while learning about teaching in a similarly 
configured online learning environment. The code Experiencing being an online 
student and its four subcodes had 28 quotations that specifically highlighted that the 
way the course was purposefully structured to position participants as online students 
had an impact on later practice. This code is also associated with three other codes 
which were populated with quotations that can be seen to relate in part to participants’ 
experience in the course as students, but primarily highlight different elements of the 
COAT course, some of which were expected results that aligned with the COAT project’s 
planned outcomes, and others which were unexpected outcomes. An expected outcome 
was for participants to reflect on the role of an online instructor. 

Reflecting on personal teaching role as online instructors. 

This code and its two subcodes encompassed 24 quotations that focused on how taking 
the COAT course prompted participants to reflect on their role as an instructor and to 
make changes to both the types of activities they included in their subsequent online 
courses and their presence in their courses as a result of this reflection. For five 
participants, a key takeaway from COAT was their role shifting to being a facilitator of 
the learning process. Other participants discussed how COAT had made them think 
about their readiness for teaching in the online environment, the importance of their 
response time to students, how to deal with disruptive students online, and the need for 
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setting up open-ended activities to engage students in the learning process. 

A subcode that dug deeper into the idea of student engagement grouped together eight 
quotations about providing opportunities for peer interaction. Five of these quotations 
referred to using group work/wikis; for example, 

I was personally terrified of the idea of working in a team 
online, but my COAT experience was exceptional and I 
vowed to use team experiences whenever possible in my 
online courses. This has been difficult for some of my 
students, and some of my student evaluations have been 
quite negative because of the group work I require, but 
some of the students share the wonderful experience I 
got during my group work in COAT and I think the 
potential for this experience is worth the risk and 
negativity some will maintain. I feel working in an online 
group takes the online educational experience to a whole 
different level, and really represents the best of what 
online teaching can offer. I only wish it could be a good 
experience for them all. 

This previous quote also demonstrates how this subcode/code can be viewed as being 
associated with the code Experiencing being an online student, as the experience of 
doing group work in COAT led to the participant incorporating group work into her 
subsequent courses which resulted in continued reflection on her teaching practice. 
Another subcode that is also associated with the experience of being an online student 
in COAT is Modeling online teaching. This subcode contained three quotations that 
referred to participants learning from the COAT facilitator modeling good online 
teaching practice. 

Being part of a community of learners. 

Just over half of the focus group participants (13 out of 24) identified that a positive 
benefit of taking COAT was that it provided them with the opportunity to interact with 
other instructors who were participating in the course, and these interactions played a 
part in influencing later practice (shown as the relationship contributes to in Figure 1). 
For example, one participant who had not taught online prior to COAT stated: “During 
the COAT class I appreciated learning from other students who either had taught online 
already or who were teaching in real time while taking the COAT class. Their stories and 
examples were invaluable.” Participants who had already taught online before COAT 
also found being part of a community of instructors/learners beneficial: “I agree, the 
COAT course gave me a place to talk to other teachers, to troubleshoot issues with like- 
folks. We do tend to teach in isolation.” Participants also expressed the wish that the 
community of learners had survived the end of the course. This desire to interact with 
other instructors separate from the assigned course curriculum or discussion prompts 
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was demonstrated in the focus groups with the many side discussions that took place 
between focus group participants that were not directly related to the focus group 
discussion prompts. 

Unimportant or Misleading Elements 

Focus group participants were asked to discuss any elements of the COAT course that 
were unimportant or misleading. Seven people stated that they could not identify any 
such elements, and six participants identified a variety of elements that they found less 
helpful or even misleading. There were few commonalities among these elements with 
individual participants identifying different aspects of the course ranging from student 
integrity information to the choice of textbook. Other participants made comments that 
were critical of the COAT course, though not coded as identifying unimportant or 
misleading elements. For example, constructive criticism in the form of 
recommendations was made with comments ranging from what was seen as an 
overwhelming amount of work to improvements to the course design. In addition to 
providing feedback on the COAT course, participants made suggestions on further 
training or networking opportunities that the COAT project should consider. 


Discussion 


In order to reflect on the categories and codes that emerged from the data analysis 
phase, metaphors were used to abstract the findings to a more conceptual level. 
Metaphors of immersion in a foreign/alien culture and of COAT being a pebble making 
ripples in pools of practice led to a further review of relevant literature. 

Immersion in an Online Learning Environment 

Two-thirds (16 out of 24) of focus group participants identified that their experience of 
being an online student in the COAT course had influenced their subsequent online 
teaching practice. Their comments about seeing an online course from a student’s 
perspective, experiencing frustration with the course navigation, and feeling 
overwhelmed resonated with a metaphor of how making the transition to online 
learning and teaching can be compared to living in a foreign country. Being immersed in 
a new environment and faced with a different culture can first cause frustration, 
confusion, self-doubt, and fear that can then lead to rethinking what is taken for granted 
as normal or commonplace behaviors. In a similar way, moving from a campus-based to 
an online learning environment can be a discombobulating experience that can make 
instructors question what they feel they know as truths about teaching and learning. 
Brookfield (1993) argued that “experiencing what it feels like to learn something 
unfamiliar and difficult is the best way to help teachers empathise with the emotions 
and feelings of their own learners as they begin to traverse new intellectual terrains” (p. 
21). The findings from the COAT alumni research study extends this idea to 
experiencing what it feels like to learn something in an unfamiliar and difficult 
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learning environment is the best way to help instructors approach their subsequent 
online teaching from a more student-centered perspective. 

This idea of comparing learning how to behave in new academic environments to 
immersion in new cultures is not original, and examples can be found in the literature of 
situated learning and communities of practice. For example, 

To talk about academic disciplines, professions, or even 
manual trades as communities or cultures will perhaps 
seem strange. Yet communities of practitioners are 
connected by more than their ostensible tasks. They are 
bound by intricate, socially constructed webs of belief, 
which are essential to understanding what they do 
(Geertz, 1983). The activities of many communities are 
unfathomable, unless they are viewed from within the 
culture.... In a significant way, learning is, we believe, a 
process of enculturation. (Brown, Collins, & Duguid, 

1989, p. 33) 

COAT participants, as adult learners, found that the sometimes frustrating experience of 
being situated as a student within an unfamiliar, authentic online learning environment 
contributed to them rethinking their teaching practice. 

The instructional approach of learning from the COAT facilitator modeling online 
teaching practice was informed by Bandura’s (1977) social learning theory. This 
approach aligns with cognitive apprenticeship (Brown, Collins, & Duguid, 1989). The 
idea of apprenticeship also appears in research focused on the teaching beliefs and 
practices of higher education instructors with Kane, Sandretto, and Heath (2002) 
arguing that 


As preparation for teaching, university academics can be 
said to have completed an ‘apprenticeship of 
observation’ during their years as undergraduate and 
graduate students (Lortie, 1975). Their beliefs and 
conceptions of good teaching are a result of this 
apprenticeship and a ‘trial by fire’ in the lecture theatre, 
classroom, or laboratory, (p. 199) 

Teaching online adds a new dimension to this trial by fire, and one focus group 

participant’s comments echoed the ideas in the above quotation: 

I second the point about experiencing an online class as 
a student. We all sat in the traditional classroom, so we 
know what takes place - what we like and what we didn't 
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like. An instructor who has not taken an online class is at 
a real disadvantage. 

Being situated as a student in an online course, that had content focused on online 
teaching theories and practice, allowed participants to observe the COAT facilitator 
model online teaching skills and strategies while she also participated in discussions 
about what she was doing and why she was doing certain actions. 

In addition to learning from the COAT facilitator, alumni also learned from the other 
COAT participants, especially those who had prior online teaching experience. Over half 
of the focus group participants (13 out of 24) identified that taking COAT provided them 
with the opportunity to interact with other instructors and these interactions played a 
part in influencing later practice. The code Being part of a community of peers that 
emerged from the analysis of the data in this study is not a new concept and is 
embedded in the literature on communities of practice (CoP). Hildreth and Kimble 
(2008) argued that "Teaching is a very personal and ‘individual’ activity, yet teachers 
benefit greatly from links with other teachers, both with colleagues in their own 
establishment and with colleagues in the wider teaching community" (p. x). 

Reflection on the learning that occurred in the community of COAT peers while they 
were immersed in an authentic online learning environment as students led to a review 
of the literature on situated cognition (Brown, Collins, & Duguid, 1989); CoP (Lave & 
Wenger, 1991); practice fields (Barab & Duffy, 2000, 2012); and collectives, networks, 
and groups (Dron & Anderson, 2007). Lave and Wenger (1991), in their book on 
situated learning, introduced CoP as nonacademic learning environments where novices 
learn from more experienced practitioners through legitimate peripheral participation, 
a form of apprenticeship. CoP as a concept has been applied to many fields since its 
inception, and Hildreth and Kimble (2008) considered it to have evolved into an 
“umbrella term” (p. xi) that now covers a range of group types with similar 
characteristics of being informal learning environments with voluntary membership of 
people interested in discussing practice and learning from each other while sharing 
resources and knowledge in a specific area. This knowledge is often tacit in nature. 

Andriessen (2005), in his research into the classification of knowledge community 
archetypes, concluded that “the same term of ‘community of practice’ has been applied 
to different types of communities, that is, to strategic communities, to informal 
communities and to informal networks” (p. 209). According to Dron and Anderson 
(2007) “individuals join Networks to associate with others of like interest or vocation, or 
who know more, or who would like to learn similar things” (p. 2461). They differentiate 
between groups as formal, structured, closed phenomena and networks as being 
informal, unstructured, and open. The COAT course is an example of a group and the 
COAT Facebook presence that was created by a COAT participant, independent from the 
COAT project, is an example of an emerging network. The unexpected, off-topic, side 
discussions in the focus groups were a demonstration of COAT alumni’s desire to 
network informally. 
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In a similar manner, using Barab and Duffy’s (2012) definition of educational practice 
fields, “Contexts in which learners, as opposed to legitimate participants, can practice 
the kinds of activities that they will encounter outside of school” (p. 34), COAT can be 
seen to be a type of practice field where participants practice online teaching and 
learning activities and skills separate from their real-life teaching situations. However, 
what is missing from this picture of COAT as a practice field is the reality that COAT 
participants were also already legitimate participants in communities of educators, and 
some had extensive prior online teaching experience which was demonstrated in the 
learning afforded by the community of peers. Figure 2 draws together ideas of formal, 
semistructured, and informal learning spaces in relation to the concept that emerged 
from this study of immersion in an online learning environment helping participants 
become enculturated into the practice of online teaching and learning. In Figure 2, the 
left oval represents COAT participants being positioned as students in an authentic, 
structured, facilitator-led training course. Within that learning space, opportunities 
were designed for semistructured peer-to-peer discussions in the Cyber Cafe where 
colleagues, with varying levels of experience of online teaching, gathered around topics 
of their own choosing. Experiences with elements within the formal COAT course and 
the semistructured internal network of peers were identified as influencing later 
teaching practice. The area to the right of Figure 2 represents informal learning spaces 
that COAT alumni, as practitioners in the workplace and as members of communities of 
online educators, may choose to join to network and contribute to communities of 
practice. 




COAT Facebook 


Community of Practice 


Informal Network 


Practitioner 


Workplace/Professional 

Organizations 



Figure 2. Networks of practice. 
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The first principle for designing training for online instructors emerged from the 
concept of being immersed in an unfamiliar online learning environment: 

Training for online instructors should be designed using 
a situated learning perspective that positions instructors 
as students in an authentic learning environment that is 
similar to the targeted teaching environment. 

The recommendations that participants made for COAT to consider training for online 
teaching in nonLMS learning environments resonate with this design principle as the 
authentic learning environment could include any number of new and emerging 
learning technologies and social media. It is important to stress that all the design 
principles that emerged from this study are situational principles for designing faculty 
training. “Situational principles are ones that are not universal - they only apply in 
some situations. They exist on a continuum from situations that are very common (close 
to universal) to ones that are highly local (apply very rarely)” (Reigeluth & Carr- 
Chellman, 2009, p. 57). By including the voices of the diverse professionals who 
participated in COAT, it is hoped that practitioners working in similar situations to the 
instructors who participated in this study will find these design principles useful and 
possibly transferable to their teaching and learning environments. 

Pools of Practice 

All 24 focus group participants identified at least one element of COAT that influenced 
their subsequent online teaching practice. A primary influence was on course (re)design 
with 18 focus group participants making comments on how taking COAT had influenced 
their subsequent online course development, design, and redesign. Research 
participants also identified other influential elements of COAT such as learning about 
LMS features, pedagogy, online instructor role, and so on. In addition, other influences 
on online teaching practice were identified such as prior teaching experiences, other 
professional development opportunities, and institutional input. Using the metaphor of 
the COAT course being one of many pebbles making ripples in individual pools of 
practice helped to conceptualize some of the ideas captured in the data analysis. In the 
same way that a pebble is a concrete object with defined edges, COAT is a specific course 
with defined learning outcomes. The number and type of ripples from a pebble being 
thrown into a pool may be expected or unexpected depending on the situational 
circumstances. The impact of COAT on an individual’s practice may align with the 
defined learning outcomes of the training, but may be unexpected depending on the 
instructor and the teaching context. Figure 3 portrays that a training course such as 
COAT is one of many possible influences on the professional practice of individual 
instructors and that the ripples from a training course are diverse, specific to a 
particular teaching context, and may be unanticipated. Figure 3 shows some examples 
of other possible influences on practice, but more pebbles are possible depending on the 
instructor’s prior and current learning and work-related experiences. 
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Figure 3. Individual pool of practice. 


(Re)designing online courses. 

COAT was designed to prepare instructors to teach predesigned courses, and, although 
basic instructional design principles were introduced in the COAT course, the emphasis 
of the COAT content was on the delivery, not on the design of online courses. Some 
participants in this research study were disappointed with the lack of emphasis on 
designing courses. The data collected in this study highlighted that for many instructors 
course design issues were one of their major concerns, as they are asked to design and 
then teach online courses with limited, and sometimes nonexistent, prior online 
teaching and instructional design experience or institutional support. Second, 
instructors have an almost implicit responsibility to customize the course to meet the 
divergent and particular needs of students, thus many both want to and feel a 
responsibility to edit and improve existing courses that they are hired to teach. The 
assumption of the COAT project that participants would teach courses predesigned by 
teams proved to be incorrect. The findings demonstrated that COAT’s impact was 
broader than planned with an unexpected outcome being that a key takeaway from the 
COAT course was its impact on participants’ (re)designing online courses which 
highlighted the need for attention within the COAT project for offering optional skill 
development in course design. Participants in this study made suggestions that COAT 
could consider on how course design could be further explored either in the current 
course or in potential advanced courses. 
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Other professional practice. 

Another unexpected outcome of participating in the COAT course was the impact the 
training had on professional practices other than online teaching practice. Four focus 
group participants commented that COAT had affected their campus-based teaching 
practice. In addition to COAT’s impact on campus-based teaching, six focus group 
participants said COAT had impacted their nonteaching practice in the areas of 
instructional design, managing online programs, and training faculty. 

The second principle for designing training for online instructors emerged from the 
findings on unexpected outcomes from taking COAT: 

Training for online instructors should prepare 
participants for diverse teaching situations which might 
include requirements to (re)design online courses and 
opportunities to teach in emerging learning 
environments. 


Conclusion 


The COAT project originated in the desire of a group of instructional designers, online 
faculty, and administrators from various institutions to collaboratively tackle the 
growing problem of how to best provide quality, accessible training for instructors who 
are making the transition to online teaching. Using a DBR methodological approach 
within an overall interpretivist research paradigm, this study evaluated whether the 
content, structure, and instructional approaches of the COAT course effectively helped 
instructors teach their subsequent online courses. Research participants identified that 
the experience of being situated as students in an authentic online course focused on 
online teaching and learning positively influenced their later online teaching, campus- 
based teaching, and nonteaching professional practice. This study provided detailed 
feedback for the COAT project, and the design principles that emerged from this study 
may be of interest to researchers and professionals who are involved in developing 
training for instructors who teach online. The findings from this study expand 
knowledge and contribute to the research literature on training for both experienced 
and inexperienced online instructors. Other recent studies focused on training for 
online instructors have resulted in similar findings to this study (see, for example, 
Eliason & Holmes, 2010; Kukulska-Hulme, 2012; MacDonald, 2010; McQuiggan, 2011; 
Storandt, Dossin, & Piacentini Lacher, 2012; Terantino & Agbehonou, 2012). By 
combining the findings from these research studies that were conducted in different 
contexts, the resulting design principles become more grounded in diverse situations 
and learning environments thus adding to the likelihood of transferability as effective 
design principles and practices to additional contexts. 
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